texts:
I'm trying to build a system which tackles the following problem. Given a list of research papers L, I would like to extract the following information from each paper:

The problem addressed by the paper
Solution to the problem proposed by the paper
(Optional) Results, i.e. how good is the solution

I have dealt with the easy part, i.e. setting up the blocks that interact with web search engines (e.g. Google scholar) and gather the data, given a set of search keys.

However, I don't have the knowledge to set up the 'intelligent' part, i.e. the one which extracts the aforementioned information from the text data. I would prefer to extract information based on the Abstract, but analysis the complete text wouldn't be a dealbreaker for me. I'm mostly interested in Computer Networking papers, e.g. published on SIGCOMM, CoNEXT, etc.

My questions are:

Are there tools - preferably open source - which can extract this information from research papers?
If not, what techniques do you recommend me to start looking at (e.g. research papers, tutorials, etc.) to accomplish this?


texts: How to start building a statistical regression analysis model with multiple categorical/discrete input variables of high dimension in Python

texts:
I'm fairly new to data science and ML. I have data of an item going through a release process. I have collected data on various variables such as "product category", "product line", "design country", "hour of day started" and I also have data on "total time" which is the time it took the item going through the entire process. In total I have 18 different input variables where each variable is either a categorical or a discrete number such as "hour of day started".

Design_cntry      Prod_category    prod_line   ...   time_minutes
   A                  A1             A11       ...     43.2
   B                  B1             A11       ...     20.1
   C                  E1             B11       ...     15.0
  ...                ...             ...       ...     ....

I want to build a statistical regression analysis model in python which outputs the probability of a statement. Say for instance P(time > 1000 min | product category = A, product line = B, ... ) and am wondering how to tackle this problem? Are there any general ways of doing this? Or good articles/literature on this topic anyone could recommend?

I only have non negative data, so maybe there are any good regression forms based on exponential distributions?


texts:
In spirit of the famous Tensorflow Fizz Buzz joke and XOr problem I started to think, if it's possible to design a neural network that implements $y = x^2$ function?

Given some representation of a number (e.g. as a vector in binary form, so that number 5 is represented as [1,0,1,0,0,0,0,...]), the neural network should learn to return its square - 25 in this case.

If I could implement $y=x^2$, I could probably implement $y=x^3$ and generally any polynomial of x, and then with Taylor series I could approximate $y=\sin(x)$, which would solve the Fizz Buzz problem - a neural network that can find remainder of the division.

Clearly, just the linear part of NNs won't be able to perform this task, so if we could do the multiplication, it would be happening thanks to activation function.

Can you suggest any ideas or reading on subject?


texts: BERT is running out of memory in forward pass for my dictionary

texts:
Running code from this answer, my BERT is running out for my 4k words dictionary.
I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly:
from transformers import BertModel, BertTokenizer

model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

encoded_inputs = tokenizer(labels, padding = True, truncation = True, return_tensors = 'pt')

ids = encoded_inputs['input_ids']
mask = encoded_inputs['attention_mask']

output = model(ids, mask)
lab_embeddings = output.last_hidden_state.tolist()

gives me memory leakage. How can I manage this with batching since I don't have labels for classification or something like that?


texts: How to deal with time series which change in seasonality or other patterns?

texts:
Background
I'm working on a time series data set of energy meter readings. The length of the series varies by meter - for some I have several years, others only a few months, etc. Many display significant seasonality, and often multiple layers - within the day, week, or year.
One of the things I've been working on is clustering of these time series. My work is academic for the moment, and while I'm doing other analysis of the data as well, I have a specific goal to carry out some clustering.
I did some initial work where I calculated various features (percentage used on weekends vs. weekday, percentage used in different time blocks, etc.). I then moved on to looking at using Dynamic Time Warping (DTW) to obtain the distance between different series, and clustering based on the difference values, and I've found several papers related to this.
Question
Will the seasonality in a specific series changing cause my clustering to be incorrect? And if so, how do I deal with it?
My concern is that the distances obtained by DTW could be misleading in the cases where the pattern in a time series has changed. This could lead to incorrect clustering.
In case the above is unclear, consider these examples:
Example 1
A meter has low readings from midnight until 8AM, the readings then increase sharply for the next hour and stay high from 9AM until 5PM, then decrease sharply over the next hour and then stay low from 6PM until midnight. The meter continues this pattern consistently every day for several months, but then changes to a pattern where readings simply stay at a consistent level throughout the day.
Example 2
A meter shows approximately the same amount of energy being consumed each month. After several years, it changes to a pattern where energy usage is higher during the summer months before returning to the usual amount.
Possible Directions

I've wondered whether I can continue to compare whole time series, but split them and consider them as a separate series if the pattern changes considerably. However, to do this I'd need to be able to detect such changes. Also, I just don't know if this is a suitable way or working with the data.
I've also considered splitting the data and considering it as many separate time series. For instance, I could consider every day/meter combination as a separate series. However, I'd then need to do similarly if I wanted to consider the weekly/monthly/yearly patterns. I think this would work, but it's potentially quite onerous and I'd hate to go down this path if there's a better way that I'm missing.

Further Notes
These are things that have come up in comments, or things I've thought of due to comments, which might be relevant. I'm putting them here so people don't have to read through everything to get relevant information.

I'm working in Python, but have rpy for those places where R is more suitable. I'm not necessarily looking for a Python answer though - if someone has a practical answer of what should be done I'm happy to figure out implementation details myself.
I have a lot of working "rough draft" code - I've done some DTW runs, I've done a couple of different types of clustering, etc. I think I largely understand the direction I'm taking, and what I'm really looking for is related to how I process my data before finding distances, running clustering, etc. Given this, I suspect the answer would be the same whether the distances between series are calculated via DTW or a simpler Euclidean Distance (ED).
I have found these papers especially informative on time series and DTW and they may be helpful if some background is needed to the topic area: http://www.cs.ucr.edu/~eamonn/selected_publications.htm


texts: Solving correlation and covariance with MySQL

texts:
I am new to data science taking my first course in Data analysis.  I am trying to figure out where to start with a problem solving the correlation and covariance with my sql.

#this is the table I'm using
mysql> SELECT surfacearea, population FROM country LIMIT 5;
+-------------+------------+
| surfacearea | population |
+-------------+------------+
|      193.00 |     103000 |
|   652090.00 |   22720000 |
|  1246700.00 |   12878000 |
|       96.00 |       8000 |
|    28748.00 |    3401200 |
+-------------+------------+
5 rows in set (0.00 sec)

# this is my formula

WITH Cov(x,y) AS
(SELECT sum((x - avgx)*(y - avgy)) /
       (count -1 * stdev(x)*stdev(y)) as covxy
 FROM R)
SELECT covxy/stddev(x)*stddev(y) as xycorrelation
FROM R, Cov

Where in the heck am I supposed to select the table from to get my x and y values?

So far here is what I have but it's erring out.  Error code 1064 syntax error:

    SELECT avg(surfacearea)AS Xavg, avg(population) AS Yavg
(SELECT Xavg * Yavg / COUNT(population) * stddev(Xavg) * stdev(Yavg) AS covxy FROM country)
SELECT covxy / stddev(surfacearea)*stddev(population) as xycorrelation
FROM country;

Another attempt

WITH country(surfacearea,population) AS
(SELECT sum(surfacearea - avg(surfacearea))*(population - avg(population)) /
((COUNT(*) -1) * stdev(surfacearea)*stdev(population)) as covxy FROM country)
SELECT covxy / (stddev(surfacearea)*stddev(population)) as xycorrelation
FROM country;


texts: Best ML technique to suggest predictor variables

texts:
In the following dataset, the first 4 columns are predictor variables and the engine running index is the response variable.
   O2 level | Cylinder pressure | Fuel Flow | Engine temp | Engine running index
       5              15               3           31                7
       2              31              18            1               88
       1              22              66            4               31
      ...            ...              ...          ...             ...

Situation: An engineer comes with a similar data set and asks a question: "What should be the setting of my O2 level, cylinder pressure etc to get the best running index?"
Now one could take a statistical approach and try figure out the answer, but I was looking to employ some machine learning techniques. The ones I've tried, i.e. Regression, are used more to predict what the Engine running index will be, rather than suggest what the settings should be, to get the best running index. The other one I was looking at was PCA, but not sure that it will give me an answer as well.
Note: the possible max value for Engine running index is not known.
Question:
Are there ML techniques to help me answer the engineer's question?


texts: Which programming languages should I use to integrate different analysis software packages?

texts:
I am currently writing a package to streamline data analysis for a research lab. There are several different analysis software packages that we use, based out of unix, matlab, and (rarely used) python. A typical data set is about 250GB (raw), and requires at least 4 different preprocessing steps before analysis. The finished product typically ends up taking up about 1TB. The goal of my package is to allow the user to pick and choose which existing package to use for each step before running the analysis, and then the program will execute it without further user intervention. Since the goal is to integrate these different packages, written in different languages, I decided to write the program in bash to make it easy to call the actual analysis scripts no matter what language they are written in.

The program is starting to come along, but it is getting very complex because of the various idiosyncratic expectations and conventions of each analysis package. I realize bash may not be the most suitable language for complex tasks, but I like that it's easy to call scripts in different languages from there, and that it's relatively simple. The program also does a lot of file handling, which bash is good at. On the other hand, I hear it's also very slow, and it gets clunky when things get more complicated.

I'm wondering if bash is the best choice for this task. Does anyone have suggestions for other languages, or combinations of languages, that might be better suited to my needs?

I should note that I am a self-taught programmer and this is my first real programming challenge. I am mostly familiar with bash, matlab, R, and a little bit of python, but I'd like to learning new things too (C maybe?). Also, this is all going to run on unix.


texts: How to find AUC metric value for keras model?

texts:
I have wanted to find AUC metric for my Keras model. Keras doesn't have any inbuilt function to measure AUC metric.  So I found that write a function which calculates AUC metric and call this function while compiling Keras model like:

from sklearn import metrics
from keras import backend as K

def auc(y_true, y_pred):
    return metrics.roc_auc_score(K.eval(y_true), K.eval(y_pred))
model.compile(loss="binary_crossentropy", optimizer='adam',metrics=['auc'])

But this doesn't work in my case. Please help me to figure out this query.

thanks


texts: Difference - Batch Training or training multiple times?

texts:
I have a question on batch learning of neural network.
A neural network learns in batches and modifies weights in every iteration. Question: If I save checkpoints after a batch, and then load the weights at a later time and train with new batch, will it be different from training both batches in 1 go?

Example: If I have a batch size of 100 and training data of 1000 points. So would it be different, in outputted checkpoint file, if I train with 9 batches (900 data points) in one go -> save checkpoint -> load checkpoint next day -> train with last batch -> save checkpoint ... Vs give all 1000 datapoints (i.e., 10 batches) -> train -> save checkpoint file ?


texts: How do I assign a title to each of the histograms I have posted at the same time?

texts:
fig, axes = plt.subplots(1,7, sharey=True)

df2.loc[df2.country == 'BR','date_created'].hist(xrot=90,bins=30,ax=axes[0],figsize=(20,4));
df2.loc[df2.country == 'CH','date_created'].hist(xrot=90,bins=30,ax=axes[1],);
df2.loc[df2.country == 'CL','date_created'].hist(xrot=90,bins=30,ax=axes[2])
df2.loc[df2.country == 'DE','date_created'].hist(xrot=90,bins=30,ax=axes[3])
df2.loc[df2.country == 'CO','date_created'].hist(xrot=90,bins=30,ax=axes[4])
df2.loc[df2.country == 'GR','date_created'].hist(xrot=90,bins=30,ax=axes[5])
df2.loc[df2.country == 'JP','date_created'].hist(xrot=90,bins=30,ax=axes[6])
plt.xticks(size = 12);


texts: Why is Spark's LinearRegressionWithSGD very slow locally?

texts:
I have been trying to run linear regression with SGD that is found in Spark mllib for some time and I am experiencing huge performance problems. All examples that I was looking have number of iterations set to 100 and say that Spark mllib can run very fast for big data.

This the code I am having problems with:

    def train(data: RDD[TrainFeature], scalerModel: StandardScalerModel): LinearRegressionModel = {
    val labeledData = data map { trainFeature =>
      LabeledPoint(trainFeature.relevance.value, toVector(trainFeature.feature, scalerModel))
    }
    labeledData.cache()

    val algorithm = new LinearRegressionWithSGD()
    algorithm.optimizer
      .setNumIterations(10)
      .setRegParam(0.01)
      .setStepSize(0.1)

    algorithm run labeledData
  }

private def toVector(feature: Feature, scalerModel: StandardScalerModel): Vector = scalerModel transform toVector(feature)

private def toVector(feature: Feature): Vector = Vectors dense feature.coordinates.toArray

I have scaled the data first and then run the algorithm to train the model. Even when I use 10 iterations it takes around 10 minutes to train model for 70,000 entries with feature vector of size 2. And the results I get are not good at all. I start getting decent results after numberOfIterations = 1000, but that would take ages.

Is it normal for linear regression with SGD to be this slow for 70,000 vectors of size 2?

My JVM min and initial memory is set to 4g.
I have tried setting the following as well (devastated try): System.setProperty("spark.executor.memory", "3g")

I am running this locally and since normal LinearRegression written in MatLab would finish the job very fast, I am wondering what am I doing wrong?

Edit: When I look at the spark UI in jobs section, I see that it is creating way too many jobs for gradient descent. Is there a way that I could tell Spark to create very little jobs - i.e. don't split data, run everything in a single thread? Maybe that can help me debug the problem further.


texts: Why normalize when all features are on the same scale?

texts:
So I'm doing the tensorflow tutorial found here:

https://www.tensorflow.org/tutorials/keras/basic_classification

Basically, my input is a [28x28] matrix (image) that I flatten to a [1x784] vector.

The tutorial then says:

  We scale these values to a range of 0 to 1 before feeding to the neural network model. For this, cast the datatype of the image components from an integer to a float, and divide by 255.

My question is why do we need to normalize in this case? My understanding is that when we have features that are on different scales, we need normalization if not the output of the model is distorted. But in this case all pixel ranges go from 0 to 255 (all features are the same scale)

I went ahead and ran it with normalization, and get an accuracy of over 85%, whereas no normalization, my accuracy falls to 10%.

Any ideas?


texts:
Google announced a new open-source library called TensorFlow Quantum(TFQ) so I am curious to know about Tensorflow quantum.

what is TensorFlow Quantum?
How it is useful with an existing TensorFlow library?
what is hybrid quantum-classical machine learning?


texts:
I have manually created a random data set around some mean value and I have tried to use gradient descent linear regression to predict this simple mean value.

I have done exactly like in the manual and for some reason my predictor coefficients are going to infinity, even though it worked for another case.

Why, in this case, can it not predict a simple 1.4 value?

clear all;
n=10000;
t=1.4;
sigma_R = t*0.001;
min_value_t = t-sigma_R;
max_value_t = t+sigma_R;
y_data = min_value_t + (max_value_t - min_value_t) * rand(n,1);
x_data=[1:10000]';

m=0
c=0
L=0.0001
epochs=1000 %iterations

for i=1:epochs
   y_pred=m.*x_data+c;
   D_m=(-2/n)*sum(x_data.*(y_data-y_pred));
   D_c=(-2/n)*sum((y_data-y_pred));
   m=m-L*D_m;
   c=c-L*D_c;
end
plot(x_data,y_data,'.')
hold on;
grid;
plot(x_data,y_pred)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%

question update:
Hello , I have tried to write down your code in the Matlab language i am more femiliar.
my feature matrices of the form NX2 [1,X_data] is called Xmat.
i followed every step in converting the code, and i get in both Theta NAN.
Where did i go wrong?

$
%%start Matlab code
n=1000;
t=1.4;
sigma_R = t*0.001;
min_value_t = t-sigma_R;
max_value_t = t+sigma_R;
y_data = min_value_t + (max_value_t - min_value_t) * rand(n,1);
x_data=[1:1000];
L=0.0001; %learning rate
%plot(x_data,y_data);
itter=1000;

theta_0=0;
theta_1=0;
theta=[theta_0;theta_1];

itter=1000;
for i=1:itter
onss=ones(1,1000);
x_mat=[onss;x_data]';
pred=x_mat*theta;
residuals = (pred-y_data);
  for k=1:2 %start theta loop
  partial=2*dot(residuals,x_mat(:,k));
  theta(k)=theta(k)-L*partial;
  end%end theta loop
end % end itteration loop
%%end matlab code
$


texts:
Here is the data:

l <- data.frame(date = c(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24))

k <- data.frame(cost = c(25,20,18,15,5,0,0,0,10,15,30,40,45,34,26,20,10,7,4,4,15,34,57,62))

m <- cbind(l,k)

ggplot(m, aes(m$date,m$cost)) + geom_line()

This is the graph I get:

What is a good prediction model? I think I can use polynomial regression  if I subset the max and min values. see image below for better understanding. (Red for max, blue for min, lines were created using paint to explain a point)

Another way, I don't know what it is called, but I think they use it to predict weather, Not sure what the formula to obtain the  blue line below would look like. (Look at graph below for better understanding)

What would be an appropriate formula to get the blue fitted line and predict the points highlighted in red??


texts: BERT training on two tasks: what is the order of tasks?

texts:
I read that BERT has been trained on two tasks: Masked Language Modeling and Next Sentence Prediction. I want to gain clarity how exactly it was done.

Was it initially trained on  Masked Language Modeling (where we predict masked token) and later on Sentence Prediction (where we predict isnext or not)? It seems like these two tasks would require different architecture. Does BERT has some common architecture for these two tasks with interchanging layer for individual tasks? Still question of ordering remains.


texts: Are SAS Data Storage Options designed for Big Data?

texts:
I understand that SAS has some data storage options in the form of

Default Storage tables or SAS datasets - in the SAS library.
Multi-dimensional Storage cubes - in the OLAP Server.
Third Party Storage - into external databases like SQL Server ,Oracle, DB2, Teradata etc.
Parallel Storage - ?

My Questions are:

Are any of these data storage options designed for moderately big data i.e. data over a million observations?

Is this quantification at all affected by the machine that SAS is
running on?
If yes,then let's imagine a computer with 4 cores and 16GB Memory.

I've worked with SAS tables before, but they were never larger than a couple hundred thousand observations.

The Parallel Storage is supposed to be a high speed alternative for processing very large data sets.

Are these data sets stored in the SAS Library / Server etc?
Can Parallel Storage help in scaling to data over a million records?

Finally, do these data storage options make SAS function like a Database Management System - like SQL Server?


texts: Vectorized method to find matching values between two columns

texts:
I'm trying to locate the most recent rows within my Dataframe that contain the same values in two separate columns.

Presently, I am doing this slowly with looping, but I suspect there's a way to cleverly use the apply method or some other vectorized function to do this faster.
My present code:

def enumerate_matching(df):
    a = list(df['A'])
    b = list(df['B'])
    matching = []

    for i in range(0, len(a)-1):
        for j in range(i+1, len(b)):
            if a[i] == b[j]:
                matching.append(i)
                matching.append(i+j)
                break
    return matching

Is there a faster method to do this?


texts: How to run two different models in single frame?

texts:
I have mask_detector.model and yolov3 social distancing weights.
I want to run them simultaneously with a single webcam stream.
how can I run them both i.e. detecting mask and social distancing model together?


texts:
I am new to data science, and the dataset I am working on is divided into train set, test set, and validation set.
However, till now I was splitting the data with train_test_split. I expected model.fit() would have an option like validation =. But, there is none.
How I can use the CV set to improve results?


texts: Is it better to use a MinMax or a Log Return normalization to predict stock price movements?

texts:
I am trying to use a LSTM model to predict d+2 and d+3 closing prices. I am not sure whether I should normalize the data

with a MixMax scaler (-1,+1)
using the log return
(P(n)-P(0))/P(0) for each sample

I have tried quite a lot of source code from Github and they don't seem to converge on any technique.


texts: python pandas optimization: filtering on text index values

texts:
I have to filter a pandas data frame by matching a complex regular expression on a text index.

The data frame is multi level indexed, and contains more than 2 million records.

The way I'm doing is:

identifiers = self._data.index.get_level_values('identifier')
filt = ... # an_array_of_np.bool_with_the_same_length_as_my_data

pattern = ... # a complex regular expression, as a string
filt = filt & np.array(identifiers.str.contains(pat=pattern, case=False, regex=True), dtype=np.bool)

... # other filterings

Unfortunately, the line beginning by filt = filt & is very slow.

I'm wondering if you have some ideas to make it faster. I guess that's because of the identifiers.str.contains

Thanks a lot!

EDIT:

Thanks @Emre

I'm not allowed to share those data, but the code below demonstrates the problem:

Step 0: 0:00:00.013527
Step 1: 0:00:00.010127
Step 2: 0:00:04.468114
Step 3: 0:00:02.109594
Step 4: 0:00:00.027437

In fact, my feeling is that we apply the regular expression on all the values of the identifiers, while I would expect that the filter applies on the possible values of the index (lost of values are reused many times).

import pandas as pd
import numpy as np
import datetime

N = 2000000
N_DVC = 10000

def getData():
    identifiers = np.random.choice(np.array([
        "need", "need: foo", "need: bar", "need: foo: bar", "foo: need", "bar: need",
        "not: need", "not: need: foo", "not: need: bar", "not: need: foo: bar", "foo: need: not", "bar: need: not",
        "need ign", "need: foo ign", "need: bar ign", "need: foo: bar ign", "foo: need ign", "bar: need ign",
        "ign need", "need: ign foo", "need: ign bar", "need: foo: ign bar",
    ]), N)
    devices = np.random.choice(np.arange(N_DVC))
    timestamps = np.random.choice(pd.date_range('1/1/2016 00:00:00', periods=60*60*24, freq='s'), N)
    x = np.random.rand(N)
    y = np.random.rand(N)
    data = pd.DataFrame({'identifier': identifiers, 'device': devices, 'timestamp': timestamps, 'x': x, 'y': y})
    data.set_index(['device', 'identifier', 'timestamp'], drop=True, inplace=True)
    return data

def filterData(data):
    # I know those regular expressions are not perfect for the example,
    # but it mimics the real expressions I have
    rexpPlus = '^(?:[^\s]+:\s)*need(?:(?::\s[^\s]+)*:\s[^\s]+)?$'
    rexpMinus = '(?::\s)(?:(?:not)|(?:ign))(?::\s)'

    tic = datetime.datetime.now()
    identifiers = data.index.get_level_values('identifier')
    print("- Step 0: %s" % str(datetime.datetime.now() - tic))

    tic = datetime.datetime.now()
    filt = np.repeat(np.False_, data.shape[0])
    print("- Step 1: %s" % str(datetime.datetime.now() - tic))

    tic = datetime.datetime.now()
    filt = filt | np.array(identifiers.str.contains(pat=rexpPlus, case=False, regex=True), dtype=np.bool)
    print("- Step 2: %s" % str(datetime.datetime.now() - tic))

    tic = datetime.datetime.now()
    filt = filt & (~np.array(identifiers.str.contains(pat=rexpMinus, case=False, regex=True), dtype=np.bool))
    print("- Step 3: %s" % str(datetime.datetime.now() - tic))

    tic = datetime.datetime.now()
    data = data.loc[filt, :]
    print("- Step 4: %s" % str(datetime.datetime.now() - tic))

    return data

if __name__ == "__main__":
    filterData(getData())


texts: Find and count similar price for each store and output the max count and its corresponding rows

texts:
My goal is:

for each item in each store,  find the number of items in the same store within $2 price difference (i.e. number of items with similar price within the same store).

Then list all the items which has the greatest number of items.

The dataset has about 100,000 records so efficient is also important. My snapshot is a small subset for implementing ideas.
The first step has got me. I have tried groupby, count, sum. None of these get me anywhere to get the desired output. I have used df.sort_values on store_id and price. Can anyone give me a hint?
Below is some sample of my data.
import pandas as pd
data = {'item_id':  ['6dd5392a9991','363a268ae1bc','fcd248a3fe97','20d197a04656','54c6463ffc87', '1b62f63eac43', '4ed99ff1bcdf', '6e19d5b8e99b','89c9b4655a9d', '16740613e6af'],
        'store_id': ['1d632be3f72c','1d632be3f72c','1d632be3f72c','1d632be3f72c','b5d61bc3e6d1','b5d61bc3e6d1','b5d61bc3e6d1','b5d61bc3e6d1','b5d61bc3e6d1','b5d61bc3e61'],
        'price':  [23.54, 20.61, 13.63, 23.69, 13.79, 14.90,  4.09, 14.30, 4.47, 4.51]
            }

df = pd.DataFrame(data)


texts: Machine Learning to predict risk of items

texts:
I'm trying to find out what I need to research and start learning to try and apply machine learning to this problem:

  In multiple offices I have 20 chairs, all of these chairs will need to have a risk assessment carried out, as the chairs become older I want to understand when they become more at risk because of other factors such as; age, usage, last time assessed and areas level of risk. I then want to be able to come up with a simple risk score of each individual chair.

I have done some research and I think it points towards using a machine learning technique such as supervised learning with a neural network but I'm uncertain. Given my scenario what should I be looking to start learning and doing?

Furthermore I believe that from my limited knowledge that providing all of the values I'm using are the same across other items I could apply and scale out this model for other items?

Further info;

As I'm familliar with Javascript I'm choosing to use synaptics.js to create a neural network and I'm going to be generating sample data inside a node server. I'm going to be using the following for my datasets that will be used as my model for learning from:

  Age (days to years), Location(low, med, high), Office(low, med, high), Last Risk Assessment Date (days-> years capped by twenty), Daily use (Low, Medium, High) and quality(low, med, high).

All of the above will be given an expected outcome manually so that I can attempt to figure out a way for the neural network to process the above and come to the same conclusion I did.

With the above labels I will be adding a weighting that will then be used to classify the risk level it should be associated to.

The factors I'm looking for the neural network to understand is that given my factors a chair with a lot of high factors will likely be placed in a higher outcome value of say 1-4 for risk factors. I'm then hoping I can use this learning to be applied to multiple other things other than chairs and increase the labels the network can use to learn from as I go along.


texts: When does cache get expired for a RDD in pyspark?

texts:
We use .cache() on RDD for persistent caching of an dataset, My concern is when this cached will be expired?.

dt = sc.parallelize([2, 3, 4, 5, 6])
dt.cache()


texts: A low-degree polynomial alternative for softmax?

texts: I want to train on a low footprint NN which gives decent accuracy on IoTs. I find that the classification using softmax is tedious and very slow. I am looking for a low-degree polynomial alternative which will do the job, is there such a function ?


texts: Is this a good approach to classify tickets which contains description and logs?

texts:
I want to classify a dataset of support tickets which mostly contain text in the description field and sometimes server logs in a separate field.

The log field is not always there but when it's present, it's a good indicator of the target class of the ticket.

I have created a CNN based classifier which can classify the tickets based on the log field, and a SVM clf with TFIDF based features for the description field.

I am thinking of adding the output probabilities of the CNN classifier in TFIDF based  SVM classifier to combine the models as a feature column.

Is there a better way to combine these models?

Is there a better way to approach this problem, without having two separate models?


texts:
I want to predict count data. In my understanding both standard classification and regression are not well suited for this. A poisson or binomial regression algorithm seems to do the trick.

I am used to doing most of my ML tasks in sklearn. But on this topic I could not find an implementation. Are there any suitable options within the python universe for this?


texts:
I'm looking for an implementation of k-modes in pyspark. I found this and this as implementations.

First, I tried implementing k-modes using the first link and faced issues. So I went ahead and tried the second implementation on github. This one worked (as in ran without errors), but I'm still unable to use it in the proper way because of no guide/usage techniques. According to the github page:

n_modes=36
partitions=10
max_iter=10
fraction = 50000 * partitions / (data.count() * 1.0)
data = data.rdd.sample(False,fraction).toDF()

method=IncrementalPartitionedKMetaModes(n_partitions = partitions, n_clusters = n_modes,max_dist_iter = max_iter,local_kmodes_iter = max_iter,
 similarity = "frequency", metamodessimilarity = "hamming")

cluster_metamodes = method.calculate_metamodes(data)

I'm not able to understand how to use this cluster_metamodes, or what it even means.

  In short, k-modes will be performed for each partition in order to
  identify a set of modes (of clusters) for each partition. Next,
  k-modes will be repeated to identify modes of a set of all modes from
  all partitions. These modes of modes are called metamodes here.

Can someone explain the concept and tell how to use this method of clustering on a pyspark dataframe of categorical values?


texts: Identifying parts of speech for individual words (not texts)

texts:
Please forgive the potential ignorance of this question. I am not a complete virgin regarding NLP:

I am trying to assign parts of speech to individual words. I realize that many words can serve as multiple parts of speech, but I am looking for the POS that a human would be most likely to interpret the word as, seeing it out of context.

Is there any database that has this info in it for a reasonably large set of words? Or, is there any large corpus with POS tags where I could feed it  words, and get frequency distributions of the POS that each word is playing in the corpus.

Thanks.


texts:
We have a problem where we have a standardized database of Medicine names.  On the other hand, there is a subset of medicine names which could have spelling mistakes, different structure or hypens, missing words etc.  There is also some metadata available, like manufacturer name, unit size etc.

Human can easily map those two database with each other.  We have used some string comparison and created some probabilistic scoring and in some cases it serves the purpose.

But lot of times we are running into lot of nuanced issues and conditions are keep getting piled up.  Is there any idea if any machine learning type of algorithm can help?  I have basic understanding of all major algorithms but yet I am drawing blank for this problem.  Simple example is mapping Epilex 300mg tab with Epilex 300 Tablet.  I can give more examples if needed.


texts:
I have two sets of newspaper articles where I train the first newspaper dataset separately to get the topics per each newspaper article.

E.g., first newspaper dataset
article_1 = {'politics': 0.1, 'nature': 0.8, ..., 'sports':0, 'wild-life':1}

Again, I train my second newspaper dataset (from a different distributor) to get the topics per each newspaper article.

E.g., second newspaper dataset (from a different distributor)
article_2 = {'people': 0.3, 'animals': 0.7, ...., 'business':0.7, 'sports':0.2}

As shown in the examples, the topics I get from the two datasets are different, thus I manually matched similar topics based on their frequent words.

I want to identify whether the two newspaper distributors publish the same news in every week.

Hence, I am interested in knowing  if there is a systematic way of comparing the topics across two corpora and measuring their similarity. Please help me.


texts: Autoencoder doesn't learn to reduce dimesions

texts:
I coded a neural network from scratch in Python. I tried it with the XOR problem and it learned correctly. So I tried to encode an Autoencoder with 3 inputs (and therefore with also 3 outputs) to reduce a color (r, g, b) in one dimension. I have normalized the data from 0 to 1 so I can use activation functions like sigmoid, relu etc. I have tried many different activation functions and learning rates, but the Autoencoder error(calculated with the mean squared error) is high (the lowest I got is 0.1), although I have trained it for more than 30,000 iterations. Did I miss something? (I think so, but is it possible to reduce a color to a size with good accuracy in the first place?)

Thank you all


texts: python print values seasonal_decomposition

texts:
I am totally beginner in Python and after using seasonal_decompose for time series decomposition result=seasonal_decompose(series, model='additive', freq=365) I got plotted results with commands result.plot() and pyplot.show(), but I cannot understand how to print this results values on screen or how to see decomposed time series values?
I have plot, but I need to have values in console or some data file.

Complete code:

from pandas import read_csv
series=read_csv('B.csv', header=0, parse_dates=[0], index_col=0, squeeze=True)
from random import randrange
from matplotlib import pyplot
from statsmodels.tsa.seasonal import seasonal_decompose
result=seasonal_decompose(series, model='additive', freq=365)
result.plot()
pyplot.show()


texts: Best practices for scaling data science / engineering teams

texts:
I am trying to find best practices for scaling data science teams, i.e find an efficient workflow/methodology to divide work between Software Engineers and Researchers working on a same product.

I’ll explain: both the SE and Researchers need the output produced by the others but they don’t necessarily have the same constraints.
- What’s important for a SE is: code maintainability, testing, CI/CD, refactoring codebase for improved development velocity,l as little branches as possible in the repository
- What’s important for a Researcher is: pace of experiments, management of experiments, journaling of experiments, model management and versioning, multiple Git branches for experimentation

How can we reconcile between both when working on the same Git repository, in a way that satisfy both stakeholders and make the work as efficient as possible ?

For example, researchers may be unhappy about a significant refactor of their experimentation script to a package that break down code to smaller bits of code, or may be frustrated by having to make sure their code does not break existing CI tests.

Can you think of interesting patterns (or point to interesting resources, books, blogs etc.) that help make the process smoother when both stakeholders are working in the same team / product ?
Thanks much.


texts: Can we think of neurons as maps between matrices?

texts:
Usually when we think about neurons, we imagine that they enact some kind of map between real numbers. For example, a neuron might take in real numbers $x_{i}$ and weight them with parameters $W_{ij}$, so that the neuron $j$ in a subsequent layer receives an input $\sum_{i}W_{ij}x_{i}$ and outputs $f(\sum_{i}W_{ij}x_{i}+b_{j})$, where $b_{j}$ is a bias and $f$ is the activation function. However, one can instead imagine a neuron that maps matrices to matrices by simply promoting the outputs $x_{i}$ and biases $b_{j}$ to matrices, so that they carry an additional index structure, e.g., $x_{i}\rightarrow x_{i}^{\alpha\beta}$. In this convention, lower indices would then indicate the locations of objects in the network and upper indices would indicate matrix elements, so that $x_{i}^{\alpha\beta}$ denotes a matrix output from neuron $i$ with matrix elements indexed by $\alpha$ and $\beta$.

This kind of structure then tells us that an input to an arbitrary neuron, i.e. $\sum_{i}W_{ij}x_{i}^{\alpha\beta}$, is just a linear combination of the matrices output by the previous layer of the network (note that the weights are still real numbers). Further, the activation function $f$ would now be a matrix function, by which I mean the output of a neuron would be defined by the Taylor series of that function evaluated on the input matrix.

Could such a scheme work, by which I mean could such a network be practically trainable? It seems to me that the answer should be yes if the activation functions in the network are well behaved. A simple example would be something like the matrix exponential function: suppose I want to compute, for a matrix $A$,

$$e^{A} = \sum_{n=0}^{\infty}\frac{1}{n!}A^{n}$$

Further, $A$ can be written as some linear combination of other matrices $A_{i}$: $A = \sum_{i}a_{i}A_{i}$, with the $a_{i}$ real numbers. If I fix $e^{A}$, shouldn't it be possible to train a network by adjusting the $a_{i}$ until a best fit is found?

This seems like a plausible idea but I'm not aware of neurons being used in this way. Has this been discussed before in the literature?

EDIT: I have been asked to explain the purpose of decomposing $A$ into the basis given by the $A_{i}$. The order of operations I have in mind is this: suppose you are given a matrix $B = f(A)$, with $f$ a nonlinear function. I can try to find a matrix $A'$ which generates a $B'$ that closely approximates $B$. But what if I want $A'$ to have a particular form? In other words, make this into a constrained optimization problem. One way to do this is to pick a basis, which I called $A_{i}$ previously, and truncate it - this forces you to find the optimal $A'$ in the restricted basis. (This has lots of possible applications in physics that I can discuss but for now I'll just claim this is an interesting thing to do.)


texts: Is LSTM or pretrained BERTForMasked LM usable for predicting changed word in a sentence using a small dataset? (2000 samples)

texts:
I have a small (2000 samples) dataset of newspaper headlines and their humorous conterparts where only one word is changed to sound silly, for example:
Original headline: Police <officer> arrested for abuse of authority
Humorous headline: Police <dog> arrested for abuse of authority
I want to train a model to predict changed sentence by the original. I am planning to implement two models for this task: one for binary tagging of input sequences (whether a word in a sentence needs to be changed) and one for predicting sentences with changed words.
Example of Model 1 input: Police officer arrested for abuse of authority
Example of Model 1 output: <no-change> <change> <no-change> <no-change> <no-change> <no-change>
Example of Model 2 input: Police <...> arrested for abuse of authority
Example of Model 2 output: Police dog arrested for abuse of authority
I am going to use a RNN/LSTM model for sequence tagging. As for the changed word prediction task, I am thinking of either using LSTM (concatenation of two parallel LSTM layers - one to run forwards on left context of word and the other to run backwards on right context) or fine-tuning BERTForMaskedLM from huggingface/transformers.
The question is whether it would be appropraite considering the small number of data, or should I switch to some other models?


texts:
I have a large multi dimensional dataset that is generated each day.
What would be a good approach to detect any kind of 'anomaly' as compared with previous days? Is this a suitable problem that could be addressed with neural networks?
Any suggestions are appreciated.
Additional information: there are no examples, so the method should detect the anomalies itself


texts: Generate pdf from jupyter notebook without code

texts:
I have jupyter notebook which contains markdown, code and outputs (graphs). I want to generate pdf from this notebook.
I tried to hide code using html code which I get from here then I tried to download as pdf but again code shows up. But when I download as html it don't show any code but again when I tried to convert html to pdf it again shows code.


texts: Execution time of the same algorithm for different runs

texts:
There is this ongoing discussion with me and my advisor.

We have a Deep Learning algorithm (does not matter which one it is, I believe, such as CNN, LSTM etc..). We are using 4 GPU Nvidia machine and we have Tensorflow GPU installed on server.

He believes that execution time of each epoch and overall runtime should be the same if we run this algorithm multiple times. We set the seed in order to have reproducible results. Every thing is set the same, number of epochs, batch size, number of batches, the same number of instances...

I do not expect the execution times of the same algorithm for different runs to be the same on Tensorflow-GPU due to the computer power, computer architecture, memory availability, and even other internal or external interruptions to the computer during the computation. It depends heavily on a lot of secondary elements, not on the algorithm itself.  These are all my assumptions.

When I plot the overall run time, every time it is different. I cannot find proper references regarding of this issue. What do you think about it?


texts:
I have trouble understanding the masks in PixelCNN 1.

Here for A, each channel is connected to prior channels. For B, channels are also connected to themselves. My problem is in first layer when I go from 3 channels to number of filters. Let's say I have 128 filters, how is the mask then implemented? What I implemented so far with the assumption that there is same number of channels in each layer is:

H = KERNEL_SIZE[0]
W = KERNEL_SIZE[1]
N = OUT_CHANNEL
M = IN_CHANNEL

# implement context mask
mask = np.zeros((N, M, H, W))
mask[:,:,:(H//2)] = 1
mask[:,:,(H//2),range((W//2))] = 1

# implement channel mask
if mask_type == 'A':
    mask[np.triu_indices(N, k=0, m=M)] = 0
elif mask_type == 'B':
    mask[np.triu_indices(N, k=1, m=M)] = 0


texts:
Currently I have a list of Books which I need to compare to a different list.
For example in my local list I have:

 - Harry Potter and the Philosopher's Stone | J.K. Rowling
 - Harry Potter and the Deathly Hallows | J.K. Rowling
 - Harry Potter and the Goblet of Fire | J.K. Rowling
 - Harry Potter and the Prisoner of Azkaban | J.K. Rowling

When I compare I end up having a simple operation:

Book Title + Author (local list) vs [Book Title + Author] (remote list)

I'm currently using Levenshtein distance, based on that I can find out if Book is in remote system or not.

In this case if Book1 vs Book2 result score exceeds X, I assume is the same Book.

levenshtein(Book1 Title + Author, Book2 Title + Author)

I read about Word Embeddings and before I start coding want to check if converting my text information to embeddings may improve my results.


texts: Does the number of hidden layers affect the activation function?

texts:
Suppose there's a network with N hidden layers.
There are 2 cases:

The network is deep
The network is shallow

I've been wondering how N affects choosing the activation function.
Will it affect, for example, Sigmoid more than Leaky ReLU?


texts: Sns Heatmap save as png jpeg image with full labels visible via python

texts:
I am trying to save my heat map in png format using the below code, but the labels are getting trimmed when I am saving the heatmap. Please help to keep the labels inside the figure.

    array = confusion_matrix(gts, preds, labels=class_name)
    #print(gts)
    #print(preds)
    df_cm = pd.DataFrame(array, index = class_name, columns = class_name)
    print("Confusion Matrix")
    print(df_cm)
    plt.figure(figsize = (10,7))

    sns_plot = sns.heatmap(df_cm, annot=True, cmap =sns.cm.rocket_r,linecolor='white', linewidths=1)

    results_path = 'results.png'
    #print(results_path)
    plt.savefig(results_path, dpi=400)


texts: Finding unpredictability or uncertainty in a time series

texts:
I am interested in finding a statistic that tracks the unpredictability of a time series. For simplicity sake, assume that each value in the time series is either 1 or 0. So for example, the following two time series are entirely predictable
TS1: 1 1 1 1 1 1 1 1
TS2: 0 1 0 1 0 1 0 1 0 1 0 1

However, the following time series is not that predictable:
TS3: 1 1 0 1 0 0 1 0 0 0 0 0 1 1 0 1 1 1

I am looking for a statistic that given a time series, would return a number between 0 and 1 with 0 indicating that the series is completely predictable and 1 indicating the series in completely unpredictable.

I looked at some entropy measures like Kolmogorov Complexity and Shannon entropy, but neither seem to fit my requirement. In Kolmogorov complexity, the statistic value changes depending on the length of the time series (as in "1 0 1 0 1" and "1 0 1 0" have different complexities, so its not possible to compare predictability of two time series with differing number of observations). In Shannon entropy, the order of observations didn't seem to matter.

Any pointers on what would be a good statistic for my requirement?


texts: Neural Network Model using Transfer Learning not learning

texts:
I am a beginner in Deep Learning and working on Road Crack detection using transfer learning. I am working on binary classification with two classes , crack and no crack.

My distribution of two classes is as follows:

Cracks - 600 images

No cracks - 480 images

I have used data augmentation also :

 train_generator = train_datagen.flow(trainX, trainY, batch_size=16)

 val_generator = test_datagen.flow(testX, testY, batch_size= 16)

I am using VGG16 and I have frozen the lower 4 layers like this :

vgg = vgg16.VGG16(include_top=False, weights='imagenet',
                                 input_shape=input_shape)

output = vgg.layers[-1].output

output = keras.layers.Flatten()(output)

vgg_model = Model(vgg.input, output)

for layer in vgg_model.layers[:4]:

     layer.trainable = False

After that, I added two hidden layers :

model = Sequential()
model.add(vgg_model)
model.add(Dense(256, activation='relu', input_dim=input_shape))
model.add(Dense(256, activation='relu'))
model.add(Dense(2, activation='softmax'))
model.compile(loss='binary_crossentropy',
          optimizer=Adam(lr = 1e-6),
          metrics=['accuracy'])

But after 1-2 epochs nothing seems to change, neither validation accuracy nor loss. I tried using SGD optimizer also but that also didn't help. I added more layers also but didn't have any effect on accuracy and loss.The maximum validation accuracy achieved is 62%.

I tried testing an image from my dataset, for that also model gives wrong prediction. For every test image it predicts as crack, i.e label 1.

Could someone suggest how i can improve this?
Thanks!


texts: I have an xlsx file that I wish to delete the NA values in SAS. I'm new to SAS, how do I do this?


texts: How can I save this model(s) and why is the use for “with tf.Graph().as_default()”

texts:
I have been trying to train and then compile this RL algo. My problem comes when I want to save the three models.

Here is how the neural networks are defined:

def build_networks(state_shape, action_size, learning_rate, critic_weight,
               hidden_neurons, entropy):

state_input = layers.Input(state_shape, name='frames')
advantages = layers.Input((1,), name='advantages')  # PG, A instead of G

# PG
actor_1 = layers.LSTM(input_shape = (1,4),units=hidden_neurons, return_sequences = True)(state_input)
actor_2 = layers.LSTM(input_shape = (1,4), units=hidden_neurons, return_sequences = False)(actor_1)
probabilities = layers.Dense(action_size, activation='softmax')(actor_2)

# DQN
critic_1 = layers.LSTM(input_shape = (1,4), units = hidden_neurons, return_sequences = True)(state_input)
critic_2 = layers.LSTM(input_shape = (1,4), units=hidden_neurons, return_sequences = False)(critic_1)
values = layers.Dense(1, activation='linear')(critic_2)

def actor_loss(y_true, y_pred):  # PG
    #cusom loss-do stuff

# Train both actor and critic at the same time.
actor = models.Model(
    inputs=[state_input, advantages], outputs=[probabilities, values])
actor.compile(
    loss=[actor_loss, 'mean_squared_error'],  # [PG, DQN]
    loss_weights=[1, critic_weight],  # [PG, DQN]
    optimizer=tf.keras.optimizers.Adam(lr=learning_rate))

critic = models.Model(inputs=[state_input], outputs=[values])
policy = models.Model(inputs=[state_input], outputs=[probabilities])
return actor, critic, policy

As you can see, the only NN that uses compile is the actor. How can I save the three NN for later use? Do I need to save them separate or I need to save the three NN as one? Also, when I tried to save them, I used this line:

with tf.Graph().as_default():
    test_agent.actor.save_weights("Test_agent_actor.h5")
    test_agent.critic.save_weights("Test_agent_critic.h5")
    test_agent.policy.save_weights("Test_agent_policy.h5")

But when I try to load any of them, it says that:

WARNING:tensorflow:No training configuration found in save file: the model was *not* compiled. Compile it manually.
WARNING:tensorflow:Output dense missing from loss dictionary. We assume this was done on purpose. The fit and evaluate APIs will not be expecting any data to be passed to dense.

Even when I try to use the NN in validation/test dataset, I get an error that tells me that:

InvalidArgumentError: Tensor frames:0, specified in either feed_devices or fetch_devices was not found in the Graph

Or when I try to save again the NN, it tells me something about "ragged" and the file size goes from 25MB to 1kb. Why is this? What I'm doing wrong? Should I use .hdf5 instead of .h5?

And also, I'm training all this with this part of code:

with tf.Graph().as_default():
     #do some stuff

Why is this for? Why should I use this and also use import tensorflow.keras instead of import keras?

Thank you in advance!


texts: Single machine learning algorithm for multiple classes of data : One hot encoder

texts:
I have data of the following kind:

   x1  x2  y
0  0   1  1
1  0   2  2
2  0   3  3
3  0   4  4
4  1   1  4
5  1   2  8
6  1   3  12
7  1   4  16

Is it possible to construct a single machine learning algorithm in python/scikit-learn by defining column x1 in such a way that a simple linear regression should give predict(x1=0, x2=5) = 5 and predict(x1=1, x2=5) = 20. My actual problem has multiple values of x1.

To illustrate the problem better: I have the following code with one hot encoder and it doesn't seem to give the accuracy of training the data separately.

import pandas as pd
from sklearn.linear_model import LinearRegression

# Dataframe with x1 = 0 and linear regression gives a slope of 1 as expected

df = pd.DataFrame(data=[{'x1': 0, 'x2': 1, 'y': 1},
                        {'x1': 0, 'x2': 2, 'y': 2},
                        {'x1': 0, 'x2': 3, 'y': 3},
                        {'x1': 0, 'x2': 4, 'y': 4}
                        ],
                  columns=['x1', 'x2', 'y'])

X = df[['x1', 'x2']]
y = df['y']
reg = LinearRegression().fit(X, y)
print(reg.predict(np.array([[0, 5]]))) # Output is 5 as expected

# Dataframe with x1 = 1 and linear regression gives a slope of 5 as expected

df = pd.DataFrame(data=[{'x1': 1, 'x2': 1, 'y': 4},
                        {'x1': 1, 'x2': 2, 'y': 8},
                        {'x1': 1, 'x2': 3, 'y': 12},
                        {'x1': 1, 'x2': 4, 'y': 16}
                        ],
                  columns=['x1', 'x2', 'y'])

X = df[['x1', 'x2']]
y = df['y']
reg = LinearRegression().fit(X, y)
print(reg.predict(np.array([[1, 5]]))) # Output is 20 as expected

# Combine the two data frames x1 = 0 and x1 = 1

df = pd.DataFrame(data=[{'x1': 0, 'x2': 1, 'y': 1},
                        {'x1': 0, 'x2': 2, 'y': 2},
                        {'x1': 0, 'x2': 3, 'y': 3},
                        {'x1': 0, 'x2': 4, 'y': 4},
                        {'x1': 1, 'x2': 1, 'y': 4},
                        {'x1': 1, 'x2': 2, 'y': 8},
                        {'x1': 1, 'x2': 3, 'y': 12},
                        {'x1': 1, 'x2': 4, 'y': 16}
                        ],
                  columns=['x1', 'x2', 'y'])

X = df[['x1', 'x2']]
y = df['y']
reg = LinearRegression().fit(X, y)
print(reg.predict(np.array([[0, 5]]))) # Output is 8.75
print(reg.predict(np.array([[1, 5]]))) # Output is 16.25

# use one hot encoder

df = pd.get_dummies(df, columns=["x1"], prefix=["x1"])
X = df[['x1_0', 'x1_1', 'x2']]
y = df['y']
reg = LinearRegression().fit(X, y)
print(reg.predict(np.array([[1, 0, 5]]))) # Output is 8.75
print(reg.predict(np.array([[0, 1, 5]]))) # Output is 16.25

How can I use pandas and sklearn for the combined data to get the same accuracy using one machine learning model?


texts: high accuracy on non trained class in tensorflow model

texts:
I have trained a 4 multi-class (apple_nature, apple_disease, apple_blacrot, apple_healthy) image classification algorithm using TensorFlow. However, after training, we get a good accuracy model.
The problem I am facing is when I predict tomato images it gets high accuracy on apple values, so how can I fix this issue?
Actually, it gets a high accuracy rate on non trained classes.


texts: Forecast vs Prediction: What is the difference?

texts:
I use the two terms as follows:

A prediction model gets features (which can be a time series) as input and gives a fixed-length output (might be multiple values, but "atomic" in some sense)

Examples:

Dog vs Cat
Spam vs No Spam
Weather for tomorrow
Sales of next month

A forecasting model gets a time series as input and outputs a single time series.

Examples:

Amount of airline passengers: You can predict the value for the next month, but a forecast can be over an arbitrary time frame (with deteriating quality, of course)
Weather
Sales

I've never seen textbook definitions of those terms. Do you know any source that compares the two terms?

Bonus points if you give me a good German translation for both terms. By now, I call both "Vorhersage" - I have the impression there is no other German word for it.


texts: How to Find the Average of the Input Vectors

texts:
I want to find the average of input vectors. I tried to use tf.math.reduce_mean, but it went error.

If I have to use keras.layers.Average, I have to make a list of the hidden states.

Does anyone know how to solve this?

Using tf.math.reduce_mean

# Input
inputs = Input(shape=(max_length,), name="Input")

# Embedding
embed = Embedding(input_dim=n_words+1,
                  output_dim=embedding_size,
                  input_length=max_length,
                  name="Embedding")(inputs)

# Bi-LSTM
encoder = Bidirectional(LSTM(units=hidden_state_encoder_size,
                             return_sequences=True,
                             dropout=dropout_rate,
                             name="LSTM"),
                        name="Bi-LSTM")

hidden_states = encoder(embed)

# Average
average = tf.math.reduce_mean(hidden_states)

# Outputs
outputs = Dense(n_tags,
                activation="softmax",
                name="Output")(average)

Error:

ValueError: Input 0 is incompatible with layer Output: expected min_ndim=2, found ndim=0

Using Average layer

# Input
inputs = Input(shape=(max_length,), name="Input")

# Embedding
embed = Embedding(input_dim=n_words+1,
                  output_dim=embedding_size,
                  input_length=max_length,
                  name="Embedding")(inputs)

# Bi-LSTM
encoder = Bidirectional(LSTM(units=hidden_state_encoder_size,
                             return_sequences=True,
                             dropout=dropout_rate,
                             name="LSTM"),
                        name="Bi-LSTM")

hidden_states = encoder(embed)

# Average
average = Average()(hidden_states)

# Outputs
outputs = Dense(n_tags,
                activation="softmax",
                name="Output")(average)

Error:

ValueError: A merge layer should be called on a list of inputs.


texts: training data in supervised models, case of linear regresion

texts:
I have a newbie question about the case of linear regression or other supervised models for prediction. Imagine that I have the following dataset represented by the X array:

So we have n observations and p features, when I want to train a linear regression model I need to follow the formula:

which at the end can be transformed into the following summation:

At this point I have a doubt, let´s suppose that we have 3000 registers and 10 features with one variable for prediction. So, in my linear regression model (if I want it to program from scratch and not use any library) I should apply the summation formula for the 12 features 3000 times to fit the correct weights of theta. Is it like that? Any detailed explanation would be really helpful.

Thanks


texts: Why does the error function become constant while implementing stochastic gradient descent using 2D inputs?

texts:
According to Q8,9 of HW5, Caltech's Learning from data course, we have to generate 100 test points of the form (x1,x2) and get their outputs 1/0 depending on which side of a random line they lie on.
We need to use this training data for stochastic gradient descent, using the cross-entropy error and calculate the error E_out on testing data produced in a similar fashion.
I used the following python code for the same:
def sgd(inputs, outputs, weights = np.array([0,0,0]), eeta = 0.1):
    weights.shape = (3,1)
    jj = 0
    datasets = len(outputs)

    while(True):
        temp = weights
        iterator = np.random.permutation(datasets)
        for i in iterator:
            x = inputs[i]
            y = outputs[i]
            x.shape = (3,1)
            w_x = np.dot(weights.transpose(), x)
            delta = -(y*x)/(1 + exp(y*w_x))
            weights = weights - eeta*delta

        diff_abs = np.linalg.norm(temp-weights)
        sume=0
        for i in range(datasets):
            y = outputs[i]
            x = inputs[i]
            x.shape = (3,1)
            w_x = np.dot(weights.transpose(), x)
            sume += log(1 + e**(-y*w_x))
        ein = sume/datasets

        if diff_abs<0.005:
            print(jj,'\t',ein)
            # print(ein)
            return ein
            break
        jj+=1
    return weights

def calc_eout(weights, datasets):
    weights.shape = (3,1)
    inputs = generate_inputs(datasets)
    outputs = get_outputs(inputs)
    sume = 0
    for i in range(datasets):
        y = outputs[i]
        x = inputs[i]
        x.shape = (3,1)
        w_x = np.dot(weights.transpose(), x)
        sume += log(1 + e**(-y*w_x))
    return sume/datasets

eout = []
ein = []
for i in range(100):
    print(i,'\n')
    eeta = 0.01
    datasets = 100

    weights = np.array([0,0,0])

    inputs = generate_inputs(datasets)
    outputs = get_outputs(inputs)
    weights = sgd(inputs, outputs, weights, eeta)
    e_out = calc_eout(weights, datasets)
    eout.append(e_out)

print("eout = ", sum(eout)/len(eout))

Using this code, the e_in becomes almost constant to around 35-38% which really shouldn't be the case as the data is properly linearly separable. Secondly, the answer to this question is that the E_out is around 0.1, and reaches there in around 350 epochs. While my error becomes almost constant to 0.38 in 100 epochs.
Can someone tell me what mistake I am doing? It's probably something very trivial.


texts: Scaling thousands of automated forecasts in R

texts:
Hello and thanks in advance!  I'd like some advice on a scalability issue and the best way to resolve.  I'm writing an algorithm in R to produce forecasts for several thousand entities.  One entity takes about 43 seconds to generate a forecast and upload the data to my database.  That equates to about 80+ hours for the entire set of entities and that's much too long.

I thought about running several R processes in parallel, possibly many on a few different servers, each performing forecasts for a portion of total entities.  Though that would work, is there a better way?  Can Hadoop help at all?  I have little experience with Hadoop so don't really know if it can apply.  Thanks again!


texts:
Can anyone say the cross entropy Conceptually in machine learning for selecting best features, in simple way?
I confused about that.

You do not really understand something unless you can explain it to
your grandmother.
Albert Einstein


texts: How to understand pad_sequences in keras.preprocessing?

texts: I understand now the meaning for pad_sequence but wouldn't truncate the sentence will lead to loss of data and how will the model correctly identify the sentiment with incomplete data


texts: How to combine two Deep learning model weights into one

texts:
Suppose I have these two models (model1 and model2) trained from same structured data, but different datasets:

# create and fit the LSTM network on dataset1
model = Sequential()
model.add(LSTM(4, input_shape=(look_back, 6)))
model.add(Dense(1))
optimizer = Adam(lr=0.001, decay=0.00001)
model.compile(loss='mean_squared_error', optimizer=optimizer)
model.summary()
model1 = model.fit(trainX, trainY, epochs=100, batch_size=64, verbose=2)
model1.save_weights("Model1.h5")

# create and fit the LSTM network on dataset2
model = Sequential()
model.add(LSTM(4, input_shape=(look_back, 6)))
model.add(Dense(1))
optimizer = Adam(lr=0.001, decay=0.00001)
model.compile(loss='mean_squared_error', optimizer=optimizer)
model.summary()
model2 = model.fit(trainX, trainY, epochs=100, batch_size=64, verbose=2)
model2.save_weights("Model2.h5")

How do I combine Model1.h5 and Model2.h5 to make suppose Model3.h5 which has all the attributes of Model1.h5 and Model2.h5 ?

Any help will be highly appreciated.

Thanks,


texts: What affects the magnitude of lasso penalty of a feature?

texts:
Is there a way to intuitively tell if the lasso penalty for a particular feature will be small or large?
Consider the following scenario:

Imagine we use Lasso regression on a dataset of 100 features $ [X_1, X_2, X_3, ..., X_{100}] $. Now, if we take one feature and scale up its values by a factor of 10, so for example $X_1 = X_1 * 10$, and then we fit lasso regression on the new dataset, which of the following might be true?

X1 probably would be excluded from the fitted model
X1 probably would be included in the fitted model

Someone told me that 2 is true. Somehow the operation on $X_1$ would result in less lasso penalty for $X_1$, therefore the feature is likely to be kept.

I cannot understand why.
 Can anyone tell if this whole statement is not correct to begin with or if there is some truth in it?


texts: Why do pre-trained CNNs use low image resolution?

texts:
I want to use a pre-trained convolutional network for image classification. My base data has resolutions of 500x500px up to 1000x1000px. Pre-trained architectures often expect less (between 255 and 299px in case of Googles Inception network).

Firstly: Would it potentially have a big impact to use higher resolution images? I.e. is it worthwhile investigating it?
Secondly: Does it make sense and is it possible to use a pre-trained network on low resolution and re-training the last layer/classifier with higher resolutions?


texts: How to include both origin and destination in your features?

texts:
I'm trying to predict the price of transportation for trucking freight. Two important features that I think would be of great impact are Origin and Destination. What's the best way to include that in your features?

They are categorical variables and if I encode it, the dataset will be extremely sparse. I thought about converting both features into latlong coordinates too and treat them as numerical variables.

Anyone dealt with this situation before?


texts: Delete rows of a column using another column statistics in R

texts:
I have a dataframe with the following columns:

If a year (see "year" column) has less than 10 weeks reported, I would like to delete it from the dataframe. The column "weeks_reporting" gives the order of the week in a year. How can I accomplish this task in R?


texts: XGBoost results are not invariant under monotone predictor transformations?

texts:
It is believed by many that tree-based methods are invariant under monotone transformations of the predictors. But recently I've read a paper (https://arxiv.org/pdf/1611.04561.pdf, referred to as the arxiv paper later) that says whether it's invariant depends on how the split threshold is chosen (there are three methods), and according to this paper, xgboost would be invariant under transformations because it uses the sweep left method. This is mentioned in the last paragraph on pp.2 and first paragraph on pp.3.

But when I read the original xgboost paper by Chen, the split algorithm looks much more sophisticated than any method mentioned in the arxiv paper, and it looks like it should be sensitive to transformations. I've tried xgboost on regression for a few data sets, and if I have column subsampling turned on, I do see different results with predictor transformations.

Can anyone give me some confirmation on this topic? I'm confused mostly by the arxiv paper.


texts: How to accomodate different database / schema names when writing M queries in Power BI?

texts:
I am working with Power BI and I need to setup a template with some input parameters, one of them being the name of the database to get data from.

To do this, I have setup a parameter (called project_database) so that users can specify the database to be used in an Query via the Parameter Manager in Power Query Editor.

I wrote a M Query that collects data from a database like so (with project_database being the input parameter):

let
    Source = MySQL.Database("127.0.0.1", project_database, [ReturnSingleDatabase=true]),
    working = Source{[Schema=project_database,Item="example_table"]}[Data],
in
    working

The problem is, this query will generate a result table with all the generated column names being based off of the actual database name (what project_database resolves to.)

For example, if the database name was xyz, the generated column names would be xyz.id, xyz.fielda, xyz.fieldb etc...)

How do I properly reference columns throughout the M query if I cannot rely on the schema portion of their names remaining consistent, since Power BI generates them based off of the schema name?

I attempted referencing the column names by

project_database & ".field"

But this is insanely tedious to do throughout the every M query. There must be a better approach?

Thanks!


texts: Which algorithm is used in sklearn SGDClassifier when modified huber loss is used?

texts:
The documentation says:

  The loss function to be used. Defaults to ‘hinge’, which gives a
  linear SVM. The ‘log’ loss gives logistic regression, a probabilistic
  classifier. ‘modified_huber’ is another smooth loss that brings
  tolerance to outliers as well as probability estimates.

When we use 'modified_huber' loss function, which classification algorithm is used? Is it SVM? If yes, how come it is able to give probability estimates, which is something it can't do with hinge loss?


texts:
I've got some big datasets of images (a few million each), and I would like to cluster them according to images' visual similarities. I've extracted a feature vector for each image; the space of feature representations is the one I would like to cluster in practice. However, I've had some difficulties because of the following constraints:

there's no way I can know the number of clusters in advance;
the clustering algorithm must be stable (it must not be order-dependent);
I don't want to be affected by the chaining effect (i.e. elongated clusters, where very different images could be linked through a series of small transformation - morphing);
I would like the solution to be scalable (e.g. by using Apache Spark) or, even better, incremental (online) and efficient. I guess that this last point could be addressed by a streaming clustering algorithm

Given all of these constraints, I am struggling to tackle the problem. All ready-to-go algorithms in Spark MLlib require the number of clusters beforehand. Single Linkage or DBSCAN algorithms are subjected to the chaining effect (see "density reachability" definition in density-based algorithms).
If I had an approximated scalable solution, I think I would be satisfied. I can give up on some of the above constraints if the solution is efficient. To me, it seems that scalable clustering is a hot research topic (I haven't found off-the-shelf solutions).
Of course, I'm not asking for the perfect answer, but I would like to receive some suggestions if some of you have encountered similar challenges before. I've tried some hierarchical clustering techniques, other than Single Linkage (e.g. Ward's method), but the time/space complexity (O(N^2)) required is too expensive for my datasets; furthermore, it isn't intuitive how to parallelize them.
Local Sensitive Hashing seems to hold some interesting properties, however, it seems more suitable for kNN search rather than clustering.
I read this paper: A Scalable Hierarchical Clustering Algorithm Using Spark but, unfortunately, it uses Single Linkage, thus, falling into the chaining effect, which I want to avoid.
Also, this article: Detecting image similarity using Spark, LSH, and TensorFlow seems to be pretty interesting but it doesn't give much insight into the practical clustering details.


texts: Finding the duplicate values between all columns and sort in new column with Pandas?

texts:
I have this DataFrame:
    CL1 CL2 CL3 CL4
0   a   a   b   f
1   b   y   c   d
2   c   x   d   s
3   x   s   x   a
4   s   dx  s   s
5   a   c   d   d
6   s   dx  f   d
7   d   dc  g   g
8   f   x   s   t
9   c   x   a   d
10  x   y   y   a
11  c   a   x   y
12  f   s   d   s
13  d   d   w   a

Intention:

With help Pandas

1- I wanna search and find the similar values between all columns (CL1-CL4) and sort in a new column (SIM).
2- I wanna find the non-similar values between columns and sort in another column (NON-SIM).
What i want

How can i do that? With df.pivot_table i was not successful.


texts: Improving accuracy of Text Classification

texts:
I am working on a text classification problem, the objective is to classify news articles to their corresponding categories, but in this case the categories are not very broad like, politics, sports, economics, etc., but are very closely related and in some cases even partially overlapping. This is a single category classification problem and not multi-class classification. Below is the details of the method I used.

Data Preparation -

Broke the documents in list of words.
Removed stop words, punctuations.
Performed stemming.
Replaced numerical values with '#num#' to reduce vocabulary size.
Transformed the documents into TF-IDF vectors.
Sorted all words based on their TF-IDF value and selected the top 20K words, these will be used a feature list for the classification algorithm.
Used SVM.

I have 4,500 categorized documents with 17 categories, and I used 80:20 ration for training and test dataset. I used Sklearn python library.

The best classification accuracy I have managed to get is 61% and I need it to be at least 85%.

Any help on how I can improve the accuracy would be greatly appreciated. Thanks a lot. Please let me know if you need any more details.


texts: Is GNU Octave a perfect place to code neural networks

texts: GNU Octave is used for its simplicity and compiling speed to write numerical algorithms (such as eg machine learning problems), but I wanted to know if I can also use it for faster coding of neural networks.


texts: How can I map the sample from the original feature space to the new kernel feature space? (Sk-learn)

texts:
Let's say I have a very basic SVM model, implementin sk-learn:

clf = SVC(kernel='rbf', class_weight=weights, gamma=gamma)
clf.fit(X,y)

X is the sample space with 2 features.
I plot these sample using a scatter plot:

And plot the hyperplane:

But now, I want to see how the new kernel (from radial basis function) transforms my sample into a 3D space, how would I do that?

I'm more interested in getting the new set of 3D coordinates After that, plotting in 3D is not that difficult.

Thanks!


texts:
In my nonlinear dynamics class in college, we discussed a simple perceptron with two input neurons and one output neuron that is trained on the patterns

pattern | Input | Output
   1    |  1,0  |   2
   2    |  3,1  |   6
   3    |  0,0  |   0

Solving the system of equations to determine the weights

$$w_1(1) + w_2(0) = 2$$
$$w_1(3) + w_2(1) = 6$$
$$w_1(0) + w_2(0) = 0$$

gives $w_1 = 2$ and $w_2 = 0$, which means the perceptron has made an abstraction. In more advanced neural networks, (a) how do you test for abstractions in your network and (b) how do you interpret what that abstraction means?


texts: Approach to classify blocks of time series

texts:
I am wondering if there exists an approach to classify blocks of time series, and not specifically individual time series. If so, can you point me out papers/articles/tutorials where these type of data are classified ?

For example :
We would extract time series for each country that depict these information : Human Development Index evolution, urbanization, financial situation, investments, etc. If we group all these time series we would make a block of economical information for each country.
The countries are already pre-classified to developed, underdeveloped, or semi-developed countries.


texts: Clusering based on categorical variables?

texts:
I am working on a project and currently experimenting cluster analysis. The dataset is mainly categorical variables and discrete numbers. Please pardon my poor programming skills as I am not very familiar with MathJax, but I will try to summarize the data set in words in the following three examples.

imagine column 1 is participants names of course, from column2 - 5, each column's value range from 1 (least important) - 5 (most important). so in this case, column 2-8 only have discrete data.
column 6 for example, since this is a multiple choice question. row 1 chose "nice" as an answer, however, row 2 chose "poor". In this case, we have one column that contains multiple categorical answers.
for column 7-9, this is another type of multiple choice questions. However this time, each column represents only one answer. column 7 only allows string values "true", column 8 only allows string "somewhat". So in this case, we have multiple columns that represent multiple answers of the same question.

Any ideas how to solve this? Thankful to any input!

A2       A3       A4       A5       A6       A7         A8         A9
1        4        5        4        nice     true       somewhate  false
2        4        3        1        poor     true                  false
1        5        2        1        nice                somewhate
3        2        1        5        nice     true                  false


texts:
I'm studying a binary classification task with an objective function, derived from SVM, defined so:
$\vec{\xi}' S \vec{\xi}$
with:
$y_i (f(\vec{x}_i)) >= 1 - \xi_i,  i=1..l$
and:
$\xi_i >=0, i=1..l$
$f(\vec{x}_i) = \sum_{p=1}^{QP} a_p K(\vec{x}_i, \vec{x}_p) + b$
where $QP << l$.
The point is that I don't want to use the whole kernel matrix. Instead, I use submatrices of K. With this method I was able to train a data-set with 26.000 patterns (pictures converted to 32x32 RGB values and divided by 65536), so with 1024 dimensions each (F1 score close to 1.0). But when I test the model against the validation data-set I'm not able to reach an F1 score of 0.75. The prototype project is available here.


texts: Why are axes-aligned bounding boxes used in object detection

texts:
I understand (I think) why in object detection, the result is a rectangle:
it is a simple shape that can be defined by 4 variables (2 pairs coords of opposite corners or 1 pair of coords + width and height)
So more complicated shape might require more parameters which could complicate things. But for example, what if a circle was used? There would just be 3 parameters, 1 pair of coordinates of the center + the radius. Is there is something obvious I am missing?
And still regarding the bounding box, I wonder what would happen if a 5-th parameter was added that would describe the angle of the bbox. For example, consider the iPhone on this image:

I might be thinking about it wrong, but in my head, the network could have an easier time "understanding" that we want it to detect a rotated bbox that alignes with the actual iPhone than an axes-aligned bbox. For a human, it's also easier to draw the rotated bounding box (and i would argue, more intuitive) than the axes-aligned, isn't it?
And regardless of whether it would be easier for the network, the rotated bounding box would be a more precise result of the detection.


texts: Clarification wanted for make_step function of Google's deep dream script

texts:
From https://github.com/google/deepdream/blob/master/dream.ipynb

def objective_L2(dst):          # Our training objective. Google has since release a way to load
    dst.diff[:] = dst.data      # arbitrary objectives from other images. We'll go into this later.

def make_step(net, step_size=1.5, end='inception_4c/output',
              jitter=32, clip=True, objective=objective_L2):
    '''Basic gradient ascent step.'''

    src = net.blobs['data'] # input image is stored in Net's 'data' blob
    dst = net.blobs[end]

    ox, oy = np.random.randint(-jitter, jitter+1, 2)
    src.data[0] = np.roll(np.roll(src.data[0], ox, -1), oy, -2) # apply jitter shift

    net.forward(end=end)
    objective(dst)  # specify the optimization objective
    net.backward(start=end)
    g = src.diff[0]
    # apply normalized ascent step to the input image
    src.data[:] += step_size/np.abs(g).mean() * g

    src.data[0] = np.roll(np.roll(src.data[0], -ox, -1), -oy, -2) # unshift image

    if clip:
        bias = net.transformer.mean['data']
        src.data[:] = np.clip(src.data, -bias, 255-bias)

If I understand what is going on correctly, the input image in net.blobs['data'] is inserted into the NN until the layer end. Once, the forward pass is complete until end, it calculates how "off" the blob at the end is from "something".

Questions

What is this "something"? Is it dst.data? I stepped through a debugger and found that dst.data was just a matrix of zeros right after the assignment and then filled with values after the backward pass.
Anyways, assuming it finds how "off" the result of the forward pass is, why does it try to do a backwards propagation? I thought the point of deep dream wasn't to further train the model but "morph" the input image into whatever the original model's layer represents.
What exactly does src.data[:] += step_size/np.abs(g).mean() * g do? It seems like applying whatever calculation was done above to the original image. Is this line what actually "morphs" the image?

Links that I have already read through

https://stackoverflow.com/a/31028871/2750819

I would be interested in what the author of the accepted answer meant by

  we take the original layer blob and "enchance" signals in it. What
  does it mean, I don't know. Maybe they just multiply the values by
  coefficient, maybe something else.

http://www.kpkaiser.com/machine-learning/diving-deeper-into-deep-dreams/

In this blog post the author comments next to src.data[:] += step_size/np.abs(g).mean() * g: "get closer to our target data." I'm not too clear what "target data" means here.

Note I'm cross posting this from https://stackoverflow.com/q/40690099/2750819 as I was recommended to in a comment.


texts: Should I back-calculate soft labels (probabilities) of my training set when I use probabilistic classifiers?

texts:
I have 2 datasets (D1, D2) to train 2 models (M1, M2).

M1 is a probabilistic classifier, which outputs soft labels (probabilities of a sample belonging to each class) for a binary classification problem, realized by a sklearn.svm.SVC(probability=True).
M2 is a generative model, which generates samples that belong to the desired class. The input to this model is the soft class label and some other properties.
D1 has samples, their properties and their binary labels (either 0 or 1) which serve as ground truth to train M1.
D2 has unlabeled samples and their properties. Therefore, the trained M1 model is used to label them and only then they are used to train M2.

My problem:

  I want to use D1 as input to my generative model M2. However I am
  unsure whether I should use the ground truth "hard" labels (integers 0
  or 1) along with the other properties as inputs to M2, or firstly use
  M1 to back-calculate the real-valued soft labels of D1 and feed them to M1.

I hope the diagram sheds some light on my task.


texts: Opening biosemi bdf data using MNE and biosig using python

texts:
I have a biosemi bdf of EEG data which contains 32 channel.
I've opened it using biosig, everything works great, a first list is channel and inside each list there are eeg data.
But if I open it using MNE it the first list is eeg data, and the second list (inside the list of eeg data) are two list of eeg data.

this is how I open the data using MNE

raw_file=read_raw_edf("E:\eegDATA\\256\s02_reduced.bdf",preload=True,verbose=True)

Am I missing something here?


texts: What if My Word is not in Bert model vocabulary?

texts:
I am doing NER using Bert Model. I have encountered some words in my datasets which is not a part of bert vocabulary and i am getting the same error while converting words to ids. Can someone help me in this?
Below is the code i am using for bert.
df = pd.read_csv("drive/My Drive/PA_AG_123records.csv",sep=",",encoding="latin1").fillna(method='ffill')

!wget --quiet https://raw.githubusercontent.com/tensorflow/models/master/official/nlp/bert/tokenization.py

import tensorflow_hub as hub
import tokenization
module_url = 'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2'
bert_layer = hub.KerasLayer(module_url, trainable=True)

vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()
do_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
tokenizer = tokenization.FullTokenizer(vocab_file, do_lower_case)

tokens_list=['hrct',
 'heall',
 'government',
 'of',
 'hem',
 'snehal',
 'sarjerao',
 'nawale',
 '12',
 '12',
 '9999',
 'female',
 'mobile',
 'no',
 '1155812345',
 '3333',
 '3333',
 '3333',
 '41st',
 '3iteir',
 'fillow']

max_len =25
text = tokens_list[:max_len-2]
input_sequence = ["[CLS]"] + text + ["[SEP]"]
print("After adding  flasges -[CLS] and [SEP]: ")
print(input_sequence)

tokens = tokenizer.convert_tokens_to_ids(input_sequence )
print("tokens to id ")
print(tokens)
```


texts: What if the votes for 2 classes are equal in an ensemble learning technique?

texts: Suppose in ensemble learning technique, if the number of models that predict class 1 is equal to the number of models that predict class 0. Then, which class will be decided as output?


texts: Predicting the likelihood that a prediction from a linear regression model is accurate

texts:
So to set up the problem:
I have a data set that had labeled data like colour, brand and quality as independent variables and the dependent is RRP (price).
I have made a linear regression model using this data and can predict the dependent variable using the independent variables (i am using scikitlearn so just using model.predict. This is causing me siginificant problems and I'm not sure if this is the right way to deal with this and I'm not sure if this will hinder my goal of getting accurate values for predicted variables.
Is there a way to calculate the potential accuracy of the price that is predicted? It seems to me if I ask the model to predict on brand x and quality y and the model knows brand x and the quality always produces a tight range of prices the accuracy is potentially higher?


texts: What is the purpose of a confusion matrix in a classification problem?

texts:
I am studying machine learning. After some research I understood that a typical workflow for a classification problem (after having  prepared the data) is the following:

Split data in test, train and validation sets
Train the model
Generate the confusion matrix
Analyze the metrics: accuracy, precision, recall and f1
Tune hyper-parameters based on the metric I have decided to optimize.

My question is: why do we ever need the confusion matrix? Shouldn't we already know what metric we need to optimize given the type of problem we are trying to solve?

I am asking this because, as far as I understand, if we have enough computational power, we could basically group steps 2 and 5 by applying a grid-search (which consists basically in a cross-validation for each tuning parameter) which takes as an input the metric to be measured. This would imply that you need to know the metric beforehand and furthermore you cannot get the confusion matrix.

Thanks in advance for the replies.


texts: Where the objective function of Skip-Gram Negative Sampling (SGNS) come from?

texts: In the paper of "Neural Word Embedding as Implicit Matrix Factorization", there is an objective function for Skip-Gram Negative Sampling. I wanted to know where this formula come from


texts:
I'm training a CNN network to detect relations between entities in written texts.
I am suffering from an overfitting problem, I have high accuracy and low loss at the training step, but my model can't predicate anything correctly at the test phase.
I know that my model is complex (My network contains: embedding lookup, convolution layer, operations on the output of the convolution layer represent the piecewise max pooling and dense layer ), in addition, I have a few training data (I have 13,396 sentences), with a number of classes equal to 98. That may cause the problem of overfitting but I have three epoch, in each one, I have an about 150 step, in each step there will be 50 sentences picked randomly and passed to the network to be trained on. so totally my model will be trained over ~ 22,500 sentences.
The thing is, is it possible that after just 50 steps (2500 sentences) to get accuracy = 1, and loss equal to Zero !!
Is it possible that my network Immediately face the problem of overfitting, or there may be other reasons why I am getting so?
Also, I know that even if my model is flexible, I may get hight accuracy at the training phase, but still cannot get accuracy =1! there still the irreducible error which will always make my model not reaching this.
So is it even possible to get accuracy =1, even if my network suffer from overfitting? and Could overfitting also happen because my data may have so many noises?
I have to mention that, I am training my model using Google Colab, and I have trained it several times, each time I discover there is something wrong in the code after the network finished training, I just delete the files generated in the training step and do restart runtime. then train it again.
Also, the network applies the (L2 regularization) with lambda equal to 0.0001 as follow:
 self.l2_loss = tf.contrib.layers.apply_regularization(
        regularizer=tf.contrib.layers.l2_regularizer(0.0001),
        weights_list=tf.trainable_variables()
    )

and apply a drop out regularization on the piecewise max-pooling output (context_output) with keep_prob = 0.5 as follow:
context_output = tf.nn.dropout(context_output, keep_prob)

Plus, I checked the network (I made another version of it) on another data (written in another language) and it worked fine on it, but when I changed the data, which goes through the same pre-processing method, I interfaced this problem, the original data has 556596 sentences in the training phase, but I couldn't guess if the problem happened because my data has few sentences only to be trained on, I mean perhaps I would be satisfied with the idea of overfitting if the training results were too high, and the test results were very low,  but to have accuracy in the training phase equal to 1! And in the testing stage, not even one correct sentence is predicted! This makes me wonder if the lack of data is really the real problem
Bellow results I get (it continue like that giving perfect results until reaching step 450 which is my last step )
step50, softmax_loss 0, acc 1
step100, softmax_loss 0, acc 1
step150, softmax_loss 0, acc 1
step200, softmax_loss 0, acc 1
step250, softmax_loss 0, acc 1

Bellow example of results I get form testing, where the number of sentences = 4473 (100 0.0, is How many sentences the model guessed correctly from total 100 sentences, and so on. Here the number of sentences given to the network is 510):
for all test data:
100
0.0

200
0.0

300
0.0

Could someone explain to me what is happening in my case?


texts: Does Amazon RedShift replace Hadoop for ~1XTB data?

texts:
There is plenty of hype surrounding Hadoop and its eco-system.  However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying large data sets, rather than spending time and effort building a Hadoop cluster?

Also, how does Amazon Redshift compare with Hadoop with respect to setup complexity, cost, and performance?


texts:
Needed for dating portal. I currently use  OpenFace to extract features with DNN. (I don't know whether they have a usable meaning) Then compare them to some model faces' features using euclidian distance and rate according to these values. Result is below satisfactory. I'm thinking about SVM, but I need to distinguish between a few levels (rank 1-5). OneVsAll? Not sure if I need 5 classifiers just to get the measure of similarity. Any ideas.

What have been tried:

Extracted features for a set of model faces (photos chosen by me from google); extracted features from a test set and sort photos according to average (or minimum) euclidian distance to these model features; Visual estimation: many from those which are in top 10's are noticeably not similar (and not attractive) compared to a model set
Labeled test set (according to my taste) as 1 and 0 classes (70 and 1640 samples) and CV set (14 and 194). Extracted features, trained RBF SVM, performed parameter search.


texts: How to train a recommender system to improve the customer class?

texts:
Considering the definition of a recommending system

A recommender system, is a subclass of information filtering system that seeks to predict the "rating" or "preference" a user would give to an item. They are primarily used in commercial applications. Source:Wikipedia.

it is condsidered into a retail problem having two datasets:

shop - customers characteristics
historical transaction about items with their properties.

I am wondering the following question about this kind of problems:

Question 1: By recommending products from similar stores could this algorithm guarantee that the stores are going to get more sales?
Question 2: Lets say that we know if a shop is classified into high|Medium|Low customer(we have that variable), In this case, in a general level how could we do a recommendation of products that takes into an account that we expect to convert all shops as much as possible into a high customers.


texts: How to fill missing value based on other columns in Pandas dataframe?

texts:
Suppose I have a 5*3 data frame in which third column contains missing value

1 2 3
4 5 NaN
7 8 9
3 2 NaN
5 6 NaN

I hope to generate value for missing value based rule that first product second column

1 2 3
4 5 20 <--4*5
7 8 9
3 2 6 <-- 3*2
5 6 30 <-- 5*6

How can I do it use data frame? Thanks.

How to add condition to calculate missing value like this?

if 1st % 2 == 0 then 3rd = 1st * 2nd
else 3rd = 1st + 2nd

1 2 3
4 5 20 <-- 4*5 because 4%2==0
7 8 9
3 2 5 <-- 3+2 because 3%2==1
5 6 11 <-- 5+6 because 5%2==1


texts: Best algorithm to create segment in case of categorical variables

texts:
I have a SampleData (2653 observation, 11 features) of bank transaction done in the 1-month timeframe. Download Dataset Size=250KB

I want to come up with algorithms (Single or Combine) that can segment users into different categories.

Since there are mostly categorical features involved except tx_amount, Which algorithm is best suited here, or how should I approach this problem to create user segments?


texts: Export weights (formula) from Random Forest Regressor in Scikit-Learn

texts:
I trained a prediction model with Scikit Learn in Python (Random Forest Regressor) and I want to extract somehow the weights of each feature to create an excel tool for manual prediction.

The only thing that I found is the model.feature_importances_ but it doesn't help.

Is there any way to achieve it?

def performRandomForest(X_train, y_train, X_test, y_test):

    '''Perform Random Forest Regression'''

    from sklearn.ensemble  import  RandomForestRegressor

    model  =  RandomForestRegressor()
    model.fit( X_train , y_train )

    #make predictions
    expected  = y_test
    predicted  = model.predict( X_test )

    #summarize the fit of the model
    mse  = np.mean(( predicted - expected )** 2)
    accuracy = ( model.score ( X_train , y_train ))

    return model, mse, accuracy

At the moment, I use the model.predict([features]) to do it, but I need it in an excel file.


texts: How to create a Question bank out of several Question papers?

texts:
I have many questions papers. I want to extract all the questions out of them so as to create a question bank. The issue is that all the question papers are not in the same format.
I tried using Google Cloud Vision API but it is not giving satisfactory results.
Any OCR and/or Machine Learning techniques you would recommend?


texts: Supervised or unsupervised learning for predicting energy consumption for new buildings

texts: I’m working on an model for auto dimensioning district heating pipes for new district heating areas (new customers). I have energy consumption data on hourly basis and describe data about these consumers (e.g. building year, renovation year and floor area) for a lot of district heating consumers. I want to predict dimensional load (energy consumption) for new building connected to the grid based on the describe data (e.g. building year, renovation year, floor area). I have through about using supervised or unsupervised learning, but do not know which of them that fit my needs? Is there an mode (e.g. clustering, SMV, …) that are better than other for this purpose?